Eurographics  Symposium  on  Rendering  2003,  pp.  1-10 
Per  Christensen  and  Daniel  Cohen-Or  (Editors) 


Capturing  and  Rendering  With  Incident  Light  Fields 

J.  linger,^  A.  Wenger,^  T.  Hawkins,^  A.  Gardner^  and  P.  Debevec^ 

^  Linkoping  University  Norrkoping  Visualization  and  Interaction  Studio,  Sweden 
^  University  of  Southern  California  Institute  for  Creative  Technologies,  United  States 


Abstract 

This  paper  presents  a  process  for  capturing  spatially  and  directionally  varying  illumination  from  a  real-world 
scene  and  using  this  lighting  to  illuminate  computer- generated  objects.  We  use  two  devices  for  capturing  such 
illumination.  In  the  first  we  photograph  an  array  of  mirrored  spheres  in  high  dynamic  range  to  capture  the  spatially 
varying  illumination.  In  the  second,  we  obtain  higher  resolution  data  by  capturing  images  with  an  high  dynamic 
range  omnidirectional  camera  as  it  traverses  across  a  plane.  For  both  methods  we  apply  the  light  field  technique 
to  extrapolate  the  incident  illumination  to  a  volume.  We  render  computer- generated  objects  as  illuminated  by 
this  captured  illumination  using  a  custom  shader  within  an  existing  global  illumination  rendering  system.  To 
demonstrate  our  technique  we  capture  several  spatially-varying  lighting  environments  with  spotlights,  shadows, 
and  dappled  lighting  and  use  them  to  illuminate  synthetic  scenes.  We  also  show  comparisons  to  real  objects  under 
the  same  illumination. 

Categories  and  Subject  Descriptors  (according  to  ACM  CCS):  1.3.3  [Computer  Graphics]:  Light  Fields,  Illumination, 
Image-Based  Rendering,  Reflectance  and  Shading,  Image-Based  Lighting 


1.  Introduction 

Lighting  plays  a  key  role  in  the  realism  and  visual  interest  of 
computer  generated  scenes,  and  has  motivated  many  tech¬ 
niques  for  designing  and  simulating  complex  forms  of  il¬ 
lumination  for  computer-generated  objects.  Recently,  tech¬ 
niques  have  been  proposed  that  capture  light  from  the  real 
world  to  be  used  as  the  illumination  for  computer-generated 
scenes.  Using  images  of  real-world  illumination  has  proven 
useful  for  providing  realistic  lighting  as  well  as  for  ef¬ 
fectively  integrating  computer-generated  objects  into  real- 
world  scenes.  So  far,  most  techniques  for  capturing  real- 
world  illumination  acquire  a  lighting  environment  from  a 
single  point  within  a  scene.  While  such  a  measurement 
records  the  directionally  varying  illumination  from  different 
light  sources  and  surfaces  in  a  scene,  it  does  not  capture  the 
spatially  varying  illumination  in  the  scene,  i.e.  how  the  light 
varies  from  one  location  to  another.  In  lighting  design  and 
cinematography,  spatially  varying  illumination  such  as  cast 
shadows  and  shafts  of  light  plays  an  important  role  in  the 
visual  makeup  of  artistically  designed  scenes.  This  creates  a 
need  to  be  able  to  capture  the  spatially  as  well  as  direction- 
ally  varying  illumination  within  a  scene. 

Recently,  the  light  field  and  Lumigraph  concepts  pre- 


Figure  1:  The  high  fidelity  incident  light  field  capturing  de¬ 
vice.  The  Uniq  UP-1030  camera  is  mounted  onto  the  trans¬ 
lation  stage,  allowing  it  to  move  in  the  plane  and  sample  the 
incident  light  field. 


sented  techniques  for  recording  the  spatially  varying  appear¬ 
ance  of  objects  and  environments.  The  techniques  work  by 
recording  a  two-dimensional  array  of  images  across  a  sur¬ 
face,  and  can  extrapolate  the  scene’s  appearance  to  a  three- 
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dimensional  volume  by  tracing  rays  to  the  original  capture 
surface. 

In  this  work,  we  leverage  these  techniques  to  the  capture 
of  spatially  varying  illumination,  and  thus  produce  a  tech¬ 
nique  which  allows  us  to  illuminate  computer-generated  ob¬ 
jects  with  captured  light  fields  and  Lumigraphs.  We  call  this 
capture  of  spatially  varying  illumination  an  incident  light 
field. 

In  a  manner  similar  to  the  capturing  of  traditional  light 
fields,  we  constructed  two  devices  to  acquire  an  incident 
light  field  by  taking  angular  lighting  measurements  over  a 
two-dimensional  plane.  We  then  use  this  captured  light  vol¬ 
ume  to  illuminate  a  synthetic  scene. 

2.  Background  and  Related  Work 

The  work  we  present  in  this  paper  is  based  on  two  princi¬ 
pal  ideas:  first,  that  light  in  space  can  be  captured  as  a  two- 
dimensional  array  of  images  within  an  environment,  and  sec¬ 
ond,  that  light  captured  in  a  real-world  environment  can  be 
used  to  illuminate  computer-generated  objects. 

The  way  in  which  illumination  varies  within  space  was 
described  by  Adelson  and  Bergen  ^  as  the  Plenoptic  Func¬ 
tion.  They  proposed  this  seven-dimensional  function  in  the 
form  P  =  P(0,  (|),  X,t,  Vx,  Vy.Vz),  where  P  is  defined  as  the  ra¬ 
diance  arriving  at  a  point  (\4,  Vy,  Vz)  in  the  direction  (0,  (|))  at 
time  t  with  wavelength  X.  They  noted  that  this  function  con¬ 
tains  every  omnidirectional  image  of  the  world  which  can 
be  recorded  at  any  time.  Fixing  time  and  discretizing  wave¬ 
length  to  three  spectral  integrals  for  red,  green,  and  blue,  the 
five-dimensional  function  of  (0,(|),  14,^37,14)  contains  every 
omnidirectional  image  of  every  point  in  a  space.  Levoy  and 
Hanrahan  and  Gortler  et  al.  ^  noted  that  since  the  radiance 
along  a  ray  is  constant  in  unoccluded  space,  a  2-dimensional 
array  of  images  on  a  plane  called  a  Light  Field  or  Lumigraph 
can  be  used  to  reconstruct  a  5D  plenoptic  function  from  a 
4D  dataset.  Ashdown  ^  presented  Near  Field  Photometry, 
a  method  for  capturing  exitant  light  fields  using  CCD  cam¬ 
eras.  In  our  work  we  use  the  Light  Field  technique  to  capture 
the  incident  illumination  within  a  space,  and  to  extend  this 
information  to  the  remainder  of  the  space.  To  make  maxi¬ 
mum  use  of  our  spatial  sampling,  we  use  the  depth  correction 
technique  described  by  Gortler  et  al.  ^  to  project  and  focus 
our  captured  light  onto  an  approximate  geometric  model  of 
the  scene.  Heidrich  et  al  mentions  the  idea  of  using  Light 
Fields  to  illuminate  computer  generated  scenes.  Our  work 
builds  on  the  same  idea,  but  captures  real  world  lighting 
and  utilizes  global  illumination  techniques  to  illuminate  syn¬ 
thetic  scenes.  Heidrich  et  al  used  pre-computed  light  field 
information.  Canned  Light  Sources,  for  illuminating  virtual 
objects. 

Using  images  of  incident  illumination  to  produce  realistic 
shading  was  introduced  as  the  process  of  environment  map¬ 
ping  by  Blinn  and  Newell  Miller  and  Hoffman  captured 


environment  maps  from  the  real  world  by  photographing  a 
reflective  sphere  placed  in  the  environment,  and  re-mapped 
the  sphere’s  image  onto  the  surface  of  an  object  to  show 
how  the  object  might  reflect  the  light  from  that  environment. 
They  showed  that  blurring  an  environment  map  before  ap¬ 
plying  it  to  the  surface  of  an  object  could  be  used  to  simulate 
a  variety  of  object  reflectance  properties. 

Debevec  et  al  ^  used  omnidirectional  measurements  of  in¬ 
cident  illumination  as  light  sources  within  a  global  illumina¬ 
tion  rendering  context,  and  showed  that  high  dynamic  range 
photography  techniques  (in  which  a  series  of  differently  ex¬ 
posed  digital  images  is  combined  into  a  single  radiance  im¬ 
age  covering  the  entire  dynamic  range  of  the  scene)  is  useful 
for  capturing  the  full  range  of  light  encountered  in  the  real 
world.  Used  directly,  these  illumination  capture  techniques 
record  light  only  at  a  single  point  in  space,  and  thus  do  not 
capture  how  light  directions,  colors,  and  intensities  change 
within  an  environment. 

Sato  et  al.  and  Debevec  et  al.  took  initial  steps  toward 
recording  spatially- varying  illumination  by  projecting  the 
captured  illumination  onto  a  3D  model  of  the  environment 
obtained  through  stereo  or  photogrammetry  However, 
the  technique  would  not  capture  any  significant  changes  in 
incident  illumination  such  as  harsh  shadows  or  spotlights. 
For  a  synthetic  scene,  Greger  et  al.  computed  its  Irra- 
diance  Volume,  a  3D  set  of  diffuse  environment  maps  lo¬ 
cated  in  an  evenly  spaced  lattice  within  the  scene.  With 
these  pre-computed  irradiance  measurements,  they  interac¬ 
tively  rendered  a  new  diffuse  object  moving  in  the  scene  with 
nearly  correct  incident  illumination.  Looking  at  time  instead 
of  space,  Koudelka  et  al.  recorded  time-varying  incident 
illumination,  used  to  capture  and  reproduce  the  illumination 
from  a  moving  light  source.  Yang  et  al.  and  Goldlucke  et 
al.  ^  used  digital  video  cameras  to  capture  time  varying  light 
fields. 

This  paper  begins  with  this  body  of  previous  work  to 
record  spatially  varying  real-world  illumination  which  in¬ 
cludes  harsh  shadows  and  directional  light  sources,  and  uses 
such  a  captured  field  of  illumination  to  realistically  render 
new  objects  into  this  spatially-varying  lighting. 

3.  Incident  Light  Field  Parametrization  and  Sampling 

A  light  field  describes  the  distribution  of  light  in  a  static 
scene  with  fixed  illumination.  Our  interest  in  the  light  field 
lies  in  its  application  as  lighting  information  for  rendering 
virtual  objects  with  real  world  light.  As  in  Debevec  et  al.  ^ 
we  refer  to  such  a  lighting  dataset  as  an  incident  light  field,  or 
ILF.  We  note  that  an  ILF  is  fundamentally  the  same  as  a  stan¬ 
dard  light  field;  the  term  is  meant  only  to  emphasize  that  an 
ILF  is  meant  to  capture  the  full  dynamic  range  and  direction¬ 
ality  of  the  incident  illumination,  which  is  rarely  done  for 
light  fields  meant  for  direct  viewing.  Several  light  field  pa- 
rameterizations  have  been  proposed,  including  the  two-plane 
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Figure  2:  Incident  Light  Field  Parametrization.  The  inci¬ 
dent  light  field  is  defined  by  a  reduced  plenoptic  function 
P  =  P(0,(|),  w,v),  where  (w,v)  define  the  location  of  fo  in  a 
plane  and  (0,(|))  define  the  direction  towards  the  incident 
light  with  respect  to  the  plane  normal  n. 


Figure  3:  The  mirror  sphere  array.  The  incident  light  field 
is  sampled  by  assembling  a  high-dynamic  range  image  of 
the  light  refected  from  the  plane  by  an  array  of  mirrored 
spheres. 


parameterizations  presented  by  Gortler  et  al.  ^  and  Levoy  et 
al.  1^.  Since  we  are  specifically  interested  in  capturing  in¬ 
cident  illumination,  we  base  our  light  field  parametrization 
on  a  point  (w,  v)  on  a  particular  capture  plane  n  the  set  of 
incident  illumination  directions  (0,(|))  incident  upon  11.  For 
the  unoccluded  volume  above  the  capture  plane,  the  incident 
light  field  P(0,  (|),  w,  v)  can  be  used  to  extrapolate  the  incident 
illumination  conditions  within  the  volume. 

In  this  work  we  only  consider  the  incident  light  coming 
from  the  hemisphere  above  the  normal  n  of  the  plane  Ft.  Our 
light  field  function  is  therefore  over  the  domain  of: 

P  =  P(e,(|),M,v)  ,  wherej 

(|)  and  0  are  the  direction  with  respect  to  n 
u  and  V  are  the  coordinates  in  the  plane  Ft 

The  above  parametrization  describes  the  incident  light 
field  as  a  collection  of  rays  R{s)  =  fo  -\-  s  ■  where  fo  lies  in 
the  plane  FI  with  coordinates  (w,v)  and  where  is  defined 
by  (0,  (|))  as  in  Figure  2. 

The  continuous  incident  light  field  is  discretized  by  sam¬ 
pling  the  hemisphere  of  incident  light  with  a  fixed  angular 
resolution  at  locations  (/,  j)  on  an  fz  x  m  regular  grid  in  the 
plane  Ft.  The  above  parametrization  allows  us  the  capture  a 
discrete  light  field  as  a  2D  array  of  light  probes,  where  a  sin¬ 
gle  (0,  (|))  image  captures  the  directional  information  and  the 
position  of  the  light  probe  in  the  2D  array  specifies  the  (w,  v) 
coordinates  in  the  plane. 


4.  Data  Acquisition  and  Processing 

For  capturing  incident  illumination  arriving  at  a  single  point 
in  space,  Debevec  et  al.  ^  used  a  high  dynamic  range  pho¬ 
tography  technique  to  capture  the  full  range  of  luminance 
values  in  a  real-world  scene. 

In  this  work  we  designed  and  built  two  incident  light  field 
capturing  devices,  each  based  on  combining  techniques  and 
results  from  light  field  rendering  and  image-based  lighting 
research.  Our  first  apparatus  is  closely  related  to  the  high- 
dynamic  range  light  probe  acquisition  technique  introduced 
by  Debevec  et  al.  whereas  the  second  apparatus  is  based 
more  directly  on  the  methods  employed  in  the  light  field  ren¬ 
dering  world.  In  the  following  sections  we  describe  the  ex¬ 
perimental  setup  and  the  data  processing  required  for  each 
of  the  two  ILF  capturing  devices. 

4.1.  Mirror  Sphere  Array 

The  first  device  we  built  extends  the  high-dynamic  range 
light  probe  idea  by  placing  a  series  of  mirrored  spheres  on  a 
regular  grid  and  taking  a  series  of  differently  exposed  pho¬ 
tographs  to  construct  a  high-dynamic  range  representation 
of  the  incident  light  field. 

4.1.1.  Experimental  Setup 

The  capturing  setup  consists  of  an  array  of  12  x  12  1"  diam¬ 
eter  mirror  spheres  and  a  standard  digital  camera.  As  seen  in 
Figure  3  the  mirror  spheres  are  mounted  on  a  board  which 
corresponds  to  the  plane  Ft. 

We  found  that  the  best  incident  light  field  results  were  ob¬ 
tained  by  capturing  a  nearly  orthographic  view  of  the  array 
from  a  nearly  perpendicular  direction  from  the  sphere  plane. 
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Figure  4:  Mirror  spheres  array  data  set.  The  processed  in¬ 
cident  light  field  captured  with  the  mirrored  sphere  array. 
Each  rectangle  is  a  full  latitude/longitude  map  correspond¬ 
ing  to  the  incident  light  onto  a  single  mirrored  sphere. 


We  approximate  the  orthographic  view  by  placing  the  Canon 
EOS  D30  camera  far  away  from  the  mirror  sphere  array  and 
by  using  a  telephoto  lens. 

A  benefit  of  this  technique  is  that  the  capture  process  is 
very  quick,  requiring  a  single  high-dynamic  range  image. 
Unfortunately,  a  capture  of  this  type  yields  either  poor  direc¬ 
tional  resolution  or  poor  spatial  resolution  as  the  resolution 
of  the  camera  is  limited  and  must  encompass  the  entire  grid 
of  mirror  spheres. 


Figure  5:  An  ILF  high-dynamic  range  image  series,  is  a 

sequence  of  16  images  with  increasing  exposure  time  from 
left  to  right.  The  image  series  is  used  to  reconstruct  the  high- 
dynamic  range  measurement  of  incident  illumination  for  a 
specific  location  in  the  capturing  plane. 


The  directional  resolution  is  therefore  determined  by  the  res¬ 
olution  of  the  camera  whereas  the  spatial  resolution  is  deter¬ 
mined  by  how  many  locations  are  sampled  in  the  plane. 


4.2.1.  Experimental  Setup 

The  high  fidelity  capturing  device  (Figure  1)  consists  of 
three  main  parts.  The  first  is  a  computer  controllable  cam¬ 
era  with  a  fisheye  lens,  the  second  is  a  translation  stage  onto 
which  the  camera  is  mounted,  and  the  third  is  the  software 
system  that  controls  the  device  and  processes  the  captured 
data. 


4.1.2.  Data  Processing 

The  data  processing  consists  of  two  steps.  First  the  image  se¬ 
ries  with  different  exposure  times  is  assembled  into  a  high- 
dynamic  range  image  using  techniques  presented  in  Debevec 
et  al.  .  Second,  the  individual  light  probe  subimages  are 
remapped  into  a  latitude/longitude  format.  This  is  done  with 
a  custom  software  where  the  user  outlines  the  four  corner 
spheres  of  the  mirrored  ball  array.  The  program  then  regis¬ 
ters  the  light  probe  array  image  into  a  dataset  similar  to  the 
one  as  shown  in  Figure  4. 

Our  parametrization  of  the  incident  light  field  only  consid¬ 
ers  the  hemisphere  above  the  plane  normal.  A  problem  with 
this  technique  is  that  the  spheres  interrefiect,  even  for  some 
upward-facing  directions  near  the  horizon.  Fortunately,  the 
geometry  of  spheres  placed  next  to  each  other  produces  an 
unoccluded  field  of  view  of  159.2  degrees,  which  is  nearly 
the  full  hemisphere.  To  avoid  artifacts  in  renderings,  the  low¬ 
est  10.4  degrees  near  the  horizon  should  be  discounted  from 
the  dataset.  Figure  10  shows  the  artifacts  that  can  result  from 
the  interrefiections  if  they  are  not  discounted 

4.2.  High  Fidelity  Capturing  Device 

The  second  device  we  built  addresses  the  resolution  issues  of 
the  mirror  spheres  array  device.  The  device  works  by  mov¬ 
ing  the  camera  around  in  the  plane  (as  previously  done  in 
Isaksen  et  al.  and  Levoy  et  al.  and  taking  a  single  om¬ 
nidirectional  high  dynamic  range  image  for  each  position. 


The  camera  used  is  a  Uniq  UP- 1030  2/3”  single-chip  dig¬ 
ital  color  camera  with  a  maximum  resolution  of  1024  x  1024 
pixels.  To  capture  the  entire  incident  illumination  hemi¬ 
sphere,  we  mounted  a  185°  Coastal  Optics  fisheye  c-mount 
lens  to  capture  the  ILF  images. 

The  translation  stage  which  moves  the  camera  in  the 
plane  is  a  MD-2  Dual  Stepper  Motor  System  from  Arrick 
Robotics.  The  computer-controllable  stepper  motors  make 
it  possible  to  move  the  camera  with  high  accuracy  from 
one  position  to  the  next  during  the  sampling  process.  At  its 
widest  settings,  the  translation  stage  can  move  the  camera 
around  in  a  30”  x  30”  area  with  a  spatial  resolution  of  0.01”. 

The  translation  stage  moves  the  camera  along  a  regular 
grid  with  a  user  defined  resolution.  At  each  position  on  the 
regular  grid  the  camera  takes  a  series  of  16  images  with  in¬ 
creasing  exposure  time  from  1/16384^^  of  a  second  up  to  2 
seconds  (see  Fig.  5),  from  which  a  high-dynamic  range  im¬ 
age  can  be  reconstructed. 

While  the  high  fidelity  device  can  capture  an  ILF  at  ar¬ 
bitrarily  high  spatial  resolution,  the  drawback  of  this  ap¬ 
proach  is  that  the  capture  time  increases  accordingly.  With 
our  software,  a  high  dynamic  range  incident  light  field  cap¬ 
ture  and  data  processing  with  spatial  resolution  of  30  x  30 
light  probes  takes  approximately  12  hours.  This  can  be  chal¬ 
lenging  as  the  light  field  to  be  captured  must  be  constant  the 
entire  time,  which  generally  restricts  the  technique  to  indoor 
environments. 
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Brightness  Falloff  of  Fisheye  Lens 


Angle  away  from  optical  axis  [deg] 


Figure  6:  Brightness  falloff  curves  for  the  camera/lens  sys¬ 
tem.  The  brightness  falloff  curve  for  each  color  channel  red, 
green  and  blue.  Incident  light  with  a  given  intensity  that  falls 
on  the  fisheye  lens  along  the  optical  axis  produces  a  pixel 
value  almost  twice  as  bright  as  incident  light  with  the  same 
intensity  that  falls  on  the  fisheye  lens  at  a  90°  angle  from  the 
optical  axis.  We  calibrate  for  this  effect  in  the  data  acquisi¬ 
tion. 


Figure  7:  High  fidelity  incident  light  field  data.  A  30  x  30 

spatial  resolution  and  400  x  400  directional  resolution  inci¬ 
dent  light  field  captured  by  the  high  fidelity  device. 


4.2.2.  Data  Processing 

The  data  processing  involves  building  the  high-dynamic 
range  image  for  each  camera  position,  remapping  the  fisheye 
image  to  a  latitude/longitude  image  in  (0,(|)),  and  a  bright¬ 
ness  falloff  correction  process.  The  latter  process  is  nec¬ 
essary  since  the  fisheye  lens  we  use  exhibits  a  noticeable 
brightness  falloff  for  the  pixels  far  from  the  optical  axis. 
We  calibrated  the  lens  by  moving  a  constant-intensity  dif¬ 
fuse  light  source  beginning  at  the  optical  axis  and  contin¬ 
uing  to  the  periphery  of  the  fisheye  lens,  taking  images  at 
twenty  intervals  along  the  way.  Assuming  a  radially  sym¬ 
metric  falloff  pattern,  we  extrapolated  these  readings  to  pro¬ 
duce  the  brightness  falloff  curve  shown  in  Figure  6  which 
we  used  to  correct  our  ILF  images.  Figure  7  shows  the  high- 
dynamic  range,  remapped  and  brightness  falloff  corrected 
incident  light  field  data. 

5.  Rendering  with  an  ILF 

In  this  section  we  present  our  method  for  rendering  synthetic 
objects  illuminated  by  real  world  lighting  using  the  captured 
incident  light  fields.  The  algorithm  relies  on  global  illumina¬ 
tion  for  the  light  transport  in  the  scene  and  was  implemented 
as  a  custom  material  type  in  the  Radiance  lighting  simu¬ 
lation  package. 

Our  parametrization  of  the  incident  light  field  maps  any 
ray  in  world- space  to  an  incident  ray  in  the  sampled  incident 
light  field.  During  rendering,  the  radiance  contribution  of  a 
ray  to  the  scene  is  calculated  by  spatially  interpolating  be¬ 


tween  adjacent  light  probes  and  directionally  interpolating 
within  each  of  the  light  probes  using  bilinear  interpolation 
in  a  manner  similar  to  that  in  Levoy  and  Hanrahan 

5.1.  The  General  Algorithm 

The  custom  material  type  specifies  the  location  and  orien¬ 
tation  of  the  incident  light  field  plane  in  the  scene  and  its 
spatial  dimensions.  Since  our  parametrization  of  the  incident 
light  field  is  restricted  to  the  hemisphere  above  the  normal 
h  of  the  light  field  capture  plane  Ft,  we  place  the  synthetic 
scene  in  the  volume  above  Ft  as  in  Figure  8. 

In  order  for  the  shader  to  have  an  effect  on  the  rendering 
we  need  to  cause  a  ray-surface  intersection.  We  therefore 
define  additional  geometry,  in  the  simplest  case  a  sphere  en¬ 
compassing  the  entire  local  scene,  to  which  we  apply  the 
incident  light  field  material  type.  We  refer  to  the  additional 
geometry  as  auxiliary  geometry. 

When  a  ray,  Rt  {s)  =  foi  +  ^  •  f^. ,  intersects  with  the  auxil¬ 
iary  geometry  it  potentially  corresponds  to  a  ray  that  maps 
to  an  incident  illumination  value.  Since  all  the  light  in  the 
scene  is  assumed  to  be  incident  from  the  hemisphere  above 
the  plane,  rays  directed  upwards  with  respect  to  the  plane 
will  map  to  an  incident  illumination  value  in  the  incident 
light  field.  Once  a  ray  Ri  intersects  with  the  auxiliary  geom¬ 
etry  it  is  traced  backwards  to  the  incident  light  field  plane  Ft 
(see  rays  Rq  and  Ri  in  Figure  8).  If  the  backwards -traced 
ray  Ri  intersects  with  the  incident  light  field  plane  (see  Rq 
in  Figure  8),  the  intersection  point  is  used  to  perform  the 
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Figure  8:  Outline  of  the  rendering  algorithm  .  The  direc¬ 
tional  sampling  of  the  incident  light  field  is  determined  by 
the  direction  of  the  ray  Ri,  whereas  the  spatial  sampling  of 
the  incident  light  field  is  given  by  the  intersection  of  back¬ 
wards  traced  ray  Ri  with  the  incident  light  field  plane. 


bilinear  interpolation  between  adjacent  light  probes.  If  the 
back-traced  ray  does  not  intersect  the  plane  (see  R\  in  Fig¬ 
ure  8),  its  radiance  contribution  is  approximated  using  the 
closest  spatial  sample  in  the  incident  light  held  data  set.  That 
is,  we  assume  that  the  light  held  outside  of  the  sampled  area 
is  spatially  uniform  and  continues  the  samples  at  the  bound¬ 
ary  of  the  incident  light  held.  The  original  ray  directions, 
e.g.  and  ,  are  used  to  directionally  interpolate  within 
the  previously  determined  adjacent  light  probes  around  the 
intersection  of  Ri  with  the  incident  light  held  plane.  Rays 
from  the  lower  hemisphere  do  not  contribute  to  the  radiance 
in  the  scene  (see  R2  in  Figure  8). 

As  the  adjacent  light  probes  used  for  the  spatial  interpo¬ 
lation  are  not  in  the  same  location  on  the  incident  light  held 
plane,  the  radiance  originating  from  a  specihc  point  in  the 
lighting  environment  corresponds  for  each  light  probe  to  a 
slightly  different  direction.  The  closer  the  point  of  origin  of 
the  radiance  to  the  incident  light  held  plane  the  more  the 
corresponding  directions  in  the  light  probes  deviate.  To  im¬ 
prove  the  sampling  of  the  incident  light  held  we  introduce  a 
depth  correction  process  similar  to  that  presented  by  Gortler 
et  al.  ^  into  our  rendering  algorithm.  Once  we  have  deter¬ 
mined  which  four  incident  illumination  images  need  to  be 
sampled  for  the  spatial  interpolation,  the  depth  correction  re¬ 
calculates  the  directions  such  that  the  direction  vectors  con¬ 
verge  at  the  intersection  of  Ri  with  the  auxiliary  geometry 
(Figure  9).  The  lights  in  the  scene  are  assumed  to  be  direc¬ 
tionally  smooth. 

5.2.  Depth  Correction 

With  depth  correction,  the  auxiliary  geometry  becomes  more 
than  a  means  of  intercepting  the  rendering  pipeline,  but 
rather  it  serves  as  depth  information  for  the  lighting  envi¬ 
ronment  captured  by  the  incident  light  held.  The  closer  the 


Figure  9:  The  depth  correction  process.  The  incident  illu¬ 
mination  images  we  need  to  sample  for  the  spatial  interpo¬ 
lation  are,  as  before,  determined  by  the  intersection  of  the 
backwards-traced  ray  Rq  with  the  incident  light  field  plane. 
Instead  of  just  using  the  direction  of  Rq  as  the  lookup  for  the 
directional  interpolation,  depth  correction  recalculates  the 
directions  such  that  they  converge  at  the  intersection  of  Rq 
with  the  auxiliary  geometry. 


auxiliary  geometry  approximates  the  geometry  of  the  light¬ 
ing  environment  that  the  incident  light  held  was  captured  in, 
the  more  the  depth  correction  will  improve  the  directional 
sampling,  and  thus  results  in  higher  quality  renderings  with 
fewer  bilinear  interpolation  artifacts. 

6.  Results  and  Discussion 

To  demonstrate  renderings  produced  with  the  incident  light 
held  process,  we  produced  renderings  of  collections  of 
synthetic  objects  under  our  captured  illumination  measure¬ 
ments,  and  compare  these  renderings  to  digital  images  of 
real  objects  in  these  lighting  environments.  All  of  our  ren¬ 
derings  were  generated  using  the  Radiance  system  with  our 
custom  shader. 

Figure  10  shows  two  synthetic  cubes  and  a  rehective 
sphere  rendered  using  the  incident  light  held  captured  with 
the  mirror  sphere  array  seen  in  Figure  4.  The  ILF  consists  of 
two  local  spotlights  aimed  into  the  center  of  the  sphere  array. 
The  rendering  appears  to  reproduce  this  incident  illumina¬ 
tion  correctly,  indicating  shadows  and  highlights  from  each 
of  the  sources,  as  well  as  a  sharp  shadow  across  the  orange 
box  at  the  edge  of  the  illumination.  The  spots  of  light  around 
the  horizon  of  the  synthetic  mirrored  sphere  exhibits  some  of 
the  sphere  interreflection  artifacts  mentioned  in  Section  4. 

Figure  11  shows  a  real  versus  synthetic  comparison 
for  a  lighting  environment  captured  with  the  high-fldelity 
ILF  capture  device.  The  lighting  consists  of  two  principal 
sources:  a  yellow  spotlight  aimed  at  the  center  of  the  table 
and  a  blue  light  partially  occluded  by  a  nearby  rectangular 
card. 
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Figure  11(a)  shows  a  set  of  real  objects  set  into  this  light¬ 
ing  environment,  including  a  3D  print  of  a  computer  model 
of  a  Greek  statue.  With  the  real  objects  removed,  we  cap¬ 
tured  the  illumination  at  a  spatial  resolution  of  30  x  30 
and  a  directional  resolution  of  400  x  400  pixels  for  a  total 
of  144,000,000  captured  incident  light  rays.  We  virtually 
recreated  the  configuration  of  objects  using  the  computer 
model  of  the  statue  and  spheres  on  a  virtual  table  within 
Radiance,  using  the  calibrated  ILF  camera  and  a  reflectance 
standard  to  estimate  the  surface  reflectances  of  the  objects. 
We  used  a  measured  model  of  the  room  as  the  auxiliary  ge¬ 
ometry  for  depth  correction,  and  created  a  rendering  of  the 
virtual  objects  within  the  ILF  using  a  single  ambient  bounce 
and  10,000  indirect  illumination  rays  per  pixel  for  a  total 
rendering  time  of  21  hours.  The  rendering  produced  using 
this  technique  appears  in  Figure  11(b).  The  rendering  pro¬ 
duced  a  very  similar  image  to  the  real  photograph,  including 
the  blue  light  on  the  statue’s  head  which  was  extrapolated 
from  the  ILF  based  on  the  blue  light  falling  to  the  statue’s 
left  on  the  table.  Due  to  a  slight  misalignment  of  the  ILF  with 
the  synthetic  scene,  the  blue  light  on  the  statue’s  head  be¬ 
gins  slightly  higher  on  the  statue  than  in  the  photograph.  To 
emphasize  the  importance  of  capturing  the  spatially-varying 
illumination,  we  also  rendered  the  synthetic  scene  as  if  it 
were  illuminated  entirely  by  the  lighting  measurement  taken 
by  the  ILF  capture  device  in  the  blue  area  of  the  table  seen 
in  Figure  11(c).  While  the  scene  is  still  realistically  illumi¬ 
nated,  it  clearly  does  not  match  the  real  photograph. 

For  our  final  example,  we  tested  the  spatial  resolution 
of  our  capture  device  by  placed  a  tree  branch  between  a 
small  yellow  light  source  and  the  ILF  capture  device  (Fig¬ 
ure  7).  Since  the  branch  was  placed  close  to  the  ILF  plane, 
it  produced  particularly  sharp  shadows.  We  created  a  sec¬ 
ond  scene  with  the  statue  on  a  table  under  this  illumination 
and  photographed  it  as  seen  in  Figure  12(a).  Then,  as  be¬ 
fore,  we  recreated  the  scene  and  camera  viewpoint  within 
the  computer  and  illuminated  the  scene  with  the  captured  in¬ 
cident  light  field.  The  resulting  rendering  in  Figure  12(b)  is 
again  largely  consistent  with  the  real-world  photograph,  but 
it  clearly  does  not  fully  reproduce  the  high  spatial  variation 
of  the  dappled  lighting  from  the  tree  branch.  The  reason  is 
that  the  ILF’s  spatial  resolution  of  30  x  30  is  far  less  than 
what  is  needed  to  fully  capture  the  details  of  the  lighting 
for  this  scene.  Nonetheless,  the  ILF  rendering  of  the  scene 
is  still  far  closer  to  the  photograph  than  rendering  the  scene 
with  a  single  measurement  of  scene  illumination  as  in  Figure 
12(c). 

7.  Future  Work 

The  capabilities  and  limitations  of  the  current  capturing  and 
rendering  system  suggest  several  improvements  for  future 
versions  of  the  system  in  the  capture  part  as  well  as  the  ren¬ 
dering  part. 

The  incident  light  field  capture  time  for  the  high  fidelity 


Figure  10:  Rendered  scene  with  incident  light  field  data 
from  the  mirror  sphere  array.  This  scene  is  rendered  with 
the  incident  light  field  dataset  from  Figure  4.  The  inter¬ 
reflection  artifacts  from  the  spheres  in  the  capturing  device 
can  be  seen  at  the  horizon  of  the  mirror  sphere  in  the  image. 


device  is  too  high  for  the  device  to  have  any  practical  appli¬ 
cation,  for  example  on  a  movie  set.  The  capture  time  could 
be  improved  by  using  a  better  camera  that  has  a  good  low 
light  sensitivity  and  a  high  dynamic  range,  thus  reducing  the 
number  of  images  required  for  the  high-dynamic  reconstruc¬ 
tion.  The  good  low  light  capabilities  would  reduce  the  expo¬ 
sure  time  needed  to  reconstruct  the  lower  end  of  the  high- 
dynamic  range  image.  The  capture  time  could  be  further  im¬ 
proved  by  adaptively  sampling  different  areas  of  the  plane. 
For  instance  an  irradiance  map  of  the  plane  could  be  used  to 
find  areas  with  high  spatial  variation.  The  higher  frequency 
areas  could  be  sampled  less  than  areas  with  higher  variation. 
Thus  reducing  the  overall  number  of  samples  need  to  repre¬ 
sent  the  information  of  the  incident  light  field. 

The  rendering  part  of  the  system  could  be  improved  by 
recovering  depth  information  from  the  2D  array  of  light 
probes.  This  depth  information  can  then  be  used  instead  of 
the  rough  scene  approximation  geometry  to  yield  a  much 
better  depth  correction.  We  could  further  use  the  depth  in¬ 
formation  for  a  more  efficient  spatial  interpolation  between 
light  probes.  Using  the  depth  information  we  can  determine 
when  light  sources  become  occluded  and  can  possibly  re¬ 
construct  high  frequency  spatial  variations  in  the  incident 
light  field.  Working  with  depth  information  also  relaxes  the 
assumption  that  all  the  incident  light  originates  from  a  dis¬ 
tant  scene.  The  only  requirement  remaining  is  the  space  de¬ 
scribed  by  the  incident  light  field  must  be  free  of  occluders. 
The  smarter  interpolation  further  has  the  potential  of  reduc¬ 
ing  the  number  of  samples  needed  to  reconstruct  the  contin¬ 
uous  incident  light  field,  which  in  turn  would  even  further 
reduce  the  capture  time. 

Rendering  quality  and  time  could  be  drastically  improved 


(c)  The  Eurographics  Association  2003. 


Unger  et  al  /  Capturing  and  Rendering  With  Incident  Light  Fields 


by  applying  importance  sampling  to  diffuse  bounces  based 
on  the  intensity  values  in  the  incident  light  field,  instead  of 
simply  picking  a  random  direction.  This  would  produce  less 
noisy  images  in  less  rendering  time. 

Looking  at  the  data  size  of  the  captured  incident  light 
fields,  data  eompression  might  be  very  useful  when  han¬ 
dling  incident  light  fields.  Capturing  the  single  light  probes 
as  compressed  images  might  reduce  the  rendering  time  be¬ 
cause  of  the  reduced  amount  of  data  transfer. 

Another  interesting  avenue  to  pursue  would  be  the  appli¬ 
cation  of  incident  light  fields  to  lighting  reproduetion.  Imag¬ 
ine  a  lighting  reproduction  apparatus  that  is  capable  of  pro¬ 
ducing  shadows  moving  across  an  actor’s  face  or  leaving  one 
aetor  in  shadow  while  another  is  illuminated  by  a  street  light. 
Incident  light  fields  hold  the  needed  information  to  repro¬ 
duce  such  complex  lighting  environments.  Recent  work  in 
the  area  of  lighting  reproduction  Debevec  et  al.  ^  ^  made  first 
steps  in  the  direction  of  realistic  compositing  and  relighting 
of  live  performances.  The  above  approaches  would  benefit 
significantly  from  the  additional  information  provided  by  in¬ 
cident  light  fields. 

8.  Conclusion 

In  this  paper  we  presented  a  novel  approach  for  rendering 
synthetic  objects  with  spatially- varying  real  world  illumina¬ 
tion  by  fusing  ideas  from  image-based  lighting  with  ideas 
from  light  field  rendering.  We  designed  and  built  two  devices 
to  capture  incident  light  fields,  a  simple  low  fidelity  device 
using  an  array  of  mirrored  spheres,  and  a  high  fidelity  cap¬ 
ture  using  a  computer-controlled  digital  camera  with  a  fish- 
eye  lens  mounted  to  a  computer-eontrolled  translation  stage. 
We  further  showed  how  to  render  synthetic  objects  illumi¬ 
nated  by  the  captured  incident  light  fields  using  modifica¬ 
tions  to  standard  global  illumination  techniques.  We  believe 
our  approach  offers  potential  for  creating  visually  interest¬ 
ing  and  highly  realistic  images  as  well  as  for  integrating 
computer-generated  objects  into  complex  real-world  envi¬ 
ronments. 
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(c)  Rendering  with  a  single  lighting  measurement  (c)  Rendering  with  a  single  lighting  measurement 

Figure  11:  A  scene  rendered  with  an  incident  light  field.  Figure  12:  A  second  scene  rendered  with  an  ILF.  (a)  A  real- 

(a)  A  real-world  scene  illuminated  by  spatially  varying  illu-  world  scene  illuminated  by  high-frequency  spatially  varying 

mination  (b)  A  synthetic  recreation  of  the  scene  illuminated  illumination  (b)  A  synthetic  recreation  of  the  scene  illumi- 

by  the  captured  incident  light  field,  showing  consistent  illu-  nated  a  30  x  30  incident  light  field,  showing  lower  light- 

mination  throughout  the  volume  of  the  objects,  (c)  The  same  ing  detail  (c)  The  same  synthetic  scene  rendered  with  a  sin- 

synthetic  scene  rendered  with  a  single  incident  illumination  gle  incident  illumination  measurement, 

measurement. 
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